Incline CPU Testing + CI #25

cameronrutherford · 2023-10-25T20:35:40Z

From #23, need to add same CI config to Incline, and create issue for Spack modules on Incline in CI

cameronrutherford · 2023-10-25T20:58:42Z

CMakePresets.json

+            "description": "Custom changes specific for Deception"
+        },
+        {
+            "name": "incline",


Note we will probably have to have incline-cpu vs incline-gpu, and as @nkoukpaizan might want to use other compilers, we can also have ascent-llvm vs ascent-gcc and go from there? I think keeping this name tied to a spack config is key, and making the names simple and easy to use would be nice as well.

jaelynlitz · 2023-10-26T00:49:53Z

Incline build and ctest work, but when trying make test_install it tries to compile with clang15.0.0 and throws some nvcc and cuda errors - error running locally pasted to provide some more clarity to the same error in the pipeline https://gitlab.pnnl.gov/exasgd/resolve-mirror/-/jobs/131463

[05:47 PM] [litz372@dmi01 ReSolve] $ source buildsystem/incline-env.sh 
[05:47 PM] [litz372@dmi01 ReSolve] $ cd build
[05:47 PM] [litz372@dmi01 build] $ ctest
Test project /qfs/projects/exasgd/src/jaelyn-spack/ReSolve/build
    Start 1: klu_klu_test
1/6 Test #1: klu_klu_test .....................   Passed    0.05 sec
    Start 2: matrix_test
2/6 Test #2: matrix_test ......................   Passed    0.01 sec
    Start 3: matrix_handler_test
3/6 Test #3: matrix_handler_test ..............   Passed    0.01 sec
    Start 4: vector_handler_test
4/6 Test #4: vector_handler_test ..............   Passed    0.01 sec
    Start 5: gram_schmidt_test
5/6 Test #5: gram_schmidt_test ................   Passed    0.01 sec
    Start 6: logger_test
6/6 Test #6: logger_test ......................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) =   0.12 sec
[05:47 PM] [litz372@dmi01 build] $ make test_install
/qfs/projects/exasgd/src/jaelyn-spack/ReSolve/install
/qfs/projects/exasgd/src/jaelyn-spack/ReSolve/tests/functionality/
/qfs/projects/exasgd/src/jaelyn-spack/ReSolve/install/share/examples/resolve_consumer
-- The CXX compiler identification is Clang 15.0.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rocm-5.3.0/llvm/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at /qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/cmake-3.27.7-tu2rruqvexd7ngddj2f3cac3bpiw4bnr/share/cmake-3.27/Modules/CMakeDetermineCUDACompiler.cmake:180 (message):
  Failed to find nvcc.

  Compiler requires the CUDA toolkit.  Please set the CUDAToolkit_ROOT
  variable.
Call Stack (most recent call first):
  /qfs/projects/exasgd/src/jaelyn-spack/ReSolve/install/share/resolve/cmake/ReSolveConfig.cmake:10 (enable_language)
  CMakeLists.txt:10 (find_package)


-- Configuring incomplete, errors occurred!
make[3]: *** [examples/CMakeFiles/test_install.dir/build.make:70: examples/CMakeFiles/test_install] Error 1
make[2]: *** [CMakeFiles/Makefile2:569: examples/CMakeFiles/test_install.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:576: examples/CMakeFiles/test_install.dir/rule] Error 2
make: *** [Makefile:283: test_install] Error 2

jaelynlitz · 2023-10-26T01:07:01Z

Started working on the module rebuild pipelines in the meantime... some perms issues for the service account runner: https://gitlab.pnnl.gov/exasgd/resolve-mirror/-/jobs/131472#L75

Ping Kevin Fox? Or we just make the resolve dir writeable for the exasgd group (like for exago ci)?

pelesh · 2023-10-26T01:42:58Z

Incline build and ctest work, but when trying make test_install it tries to compile with clang15.0.0 and throws some nvcc and cuda errors - error running locally pasted to provide some more clarity to the same error in the pipeline https://gitlab.pnnl.gov/exasgd/resolve-mirror/-/jobs/131463

That's my bad, @jaelynlitz. If you don't mind, please create issue and assign it to me. In the meantime, you can comment out make test_install until I fix this.

pelesh · 2023-10-26T22:57:45Z

Incline build and ctest work, but when trying make test_install it tries to compile with clang15.0.0 and throws some nvcc and cuda errors - error running locally pasted to provide some more clarity to the same error in the pipeline https://gitlab.pnnl.gov/exasgd/resolve-mirror/-/jobs/131463

I believe this issue is fixed in #34.

jaelynlitz · 2023-10-27T19:01:07Z

Incline build and test pipelines pass after a rebase on develop with fixes from @ryandanehy and @pelesh! If we want to merge this progress, I can remove the code associated with the spack rebuild pipelines and tackle those in a separate PR.

pelesh · 2023-10-27T19:10:50Z

Incline build and test pipelines pass after a rebase on develop with fixes from @ryandanehy and @pelesh! If we want to merge this progress, I can remove the code associated with the spack rebuild pipelines and tackle those in a separate PR.

I think that would be a good break point. I suggest we merge this progress, first.

buildsystem/build.sh

cameronrutherford · 2023-10-27T20:05:59Z

Ready to merge

Co-authored-by: Cameron Rutherford <cameron.rutherford@me.com>

pelesh

Will merge as soon as all tests pass.

cameronrutherford commented Oct 25, 2023

View reviewed changes

jaelynlitz force-pushed the incline-dev branch from 5d74d64 to 45bf639 Compare October 26, 2023 00:07

cameronrutherford and others added 5 commits October 27, 2023 11:51

Update spack submodule after accidental deletion...

175e0d4

Add Incline CPU build.

79cb63b

add incline CI

7fccae4

allow rebuild pipeline

a5ff220

cleanup

b76d25e

jaelynlitz force-pushed the incline-dev branch from 5420344 to b76d25e Compare October 27, 2023 18:54

remove module rebuild

ddd32b5

jaelynlitz approved these changes Oct 27, 2023

View reviewed changes

jaelynlitz changed the title ~~Draft: Incline CPU Testing + CI~~ Incline CPU Testing + CI Oct 27, 2023

jaelynlitz assigned cameronrutherford and jaelynlitz Oct 27, 2023

jaelynlitz requested review from ryandanehy and pelesh October 27, 2023 19:21

ryandanehy approved these changes Oct 27, 2023

View reviewed changes

cameronrutherford commented Oct 27, 2023

View reviewed changes

buildsystem/build.sh Outdated Show resolved Hide resolved

allow build script to detect cluster from hostname

94c60cb

cameronrutherford commented Oct 27, 2023

View reviewed changes

buildsystem/build.sh Show resolved Hide resolved

exit if my cluster not set

1fa8ee4

Co-authored-by: Cameron Rutherford <cameron.rutherford@me.com>

pelesh approved these changes Oct 27, 2023

View reviewed changes

pelesh merged commit 5950ea2 into develop Oct 27, 2023
5 checks passed

pelesh deleted the incline-dev branch December 1, 2023 18:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incline CPU Testing + CI #25

Incline CPU Testing + CI #25

cameronrutherford commented Oct 25, 2023

cameronrutherford Oct 25, 2023

jaelynlitz commented Oct 26, 2023

jaelynlitz commented Oct 26, 2023 •

edited

Loading

pelesh commented Oct 26, 2023 •

edited

Loading

pelesh commented Oct 26, 2023

jaelynlitz commented Oct 27, 2023

pelesh commented Oct 27, 2023

cameronrutherford commented Oct 27, 2023

pelesh left a comment

Incline CPU Testing + CI #25

Incline CPU Testing + CI #25

Conversation

cameronrutherford commented Oct 25, 2023

cameronrutherford Oct 25, 2023

Choose a reason for hiding this comment

jaelynlitz commented Oct 26, 2023

jaelynlitz commented Oct 26, 2023 • edited Loading

pelesh commented Oct 26, 2023 • edited Loading

pelesh commented Oct 26, 2023

jaelynlitz commented Oct 27, 2023

pelesh commented Oct 27, 2023

cameronrutherford commented Oct 27, 2023

pelesh left a comment

Choose a reason for hiding this comment

jaelynlitz commented Oct 26, 2023 •

edited

Loading

pelesh commented Oct 26, 2023 •

edited

Loading